When do generalized entropies apply? How phase space volume determines entropy 
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We show how the dependence of phase space volume Ct(N) of a classical system on its size N 
uniquely determines its extensive entropy. We give a concise criterion when this entropy is not 
of Boltzmann-Gibbs type but has to assume a generalized (non-additive) form. We show that 
generalized entropies can only exist when the dynamically (statistically) relevant fraction of degrees 
of freedom in the system vanishes in the thermodynamic limit. These are systems where the bulk 
of the degrees of freedom is frozen and is practically statistically inactive. Systems governed by 
generalized entropies are therefore systems whose phase space volume effectively collapses to a 
lower- dimensional 'surface'. We explicitly illustrate the situation for binomial processes and argue 
that generalized entropies could be relevant for self organized critical systems such as sand piles, 
for spin systems which form meta-structures such as vortices, domains, instantons, etc., and for 
problems associated with anomalous diffusion. 
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Entropy relates the number of states of a system to 
an extensive quantity, which plays a fundamental role 
in its thermodynamical description. Extensive means 
that when two initially isolated systems A and B - 
with Cl A and CIb the respective numbers of states - are 
brought in contact, the entropy of the combined system 
A + B is S(CI a +b) = S(Q A ) + S(Q B ). Extensivity is 
not to be confused with additivity which is the prop- 
erty that S(CL A CL B ) = S(Ct A ) + S(Sl B ). Both, exten- 
sivity and additivity coincide if the number of states in 
the combined system is CI a +b = CI a CIb- Clearly, for a 
non-interacting system Boltzmann-Gibbs (BG) entropy, 

Sbg[p] = Yh 9bg(jPi), with #bgO) = -xhix, is simul- 
taneously extensive and additive. By 'non-interacting' 
systems (short-range, ergodic, mixing, Markovian, ...) we 
mean CI a +b = CI a CIb- For interacting statistical systems 
this is in general not true. If phase space is only partly 
visited this means CL a +b < CL a CLb- In this case, it may 
happen that an additive entropy (such as BG) no longer 
is extensive and vice versa. With the hope to understand 
interacting statistical systems within a thermodynamical 
formalism and to ensure extensivity of entropy, so called 
generalized entropies have been introduced which usually 
assume trace form 
n 

£gen[p] = ^g(Pi) , [CI ... number of states] (1) 

i=l 

where g is some function of p. It has been shown that 
g can not assume any functional form, but generalized 
entropies of trace form are restricted to the family of 
functions 



s c,d[p] oc y^T(ri+ 1,1 - clogpi) , 



(2) 



T(., .) being the incomplete gamma function, whenever 
a minimum set of requirements on g hold [l|. These 
requirements are the first three of the four Shannon- 
Khinchin (SK) axioms @, SKI: Entropy S depends 



continuously on p (g is continuous), SK2: entropy is max- 
imal for the equi-distribution pi = 1/Cl (g is concave. In 
physical systems this represents the equi-partition prin- 
ciple in micro-canonical ensembles), SK3: adding a zero- 
probability state to a system, Cl + 1 with pn+i = 0, does 
not change the entropy (g(0) = 0), and SK4: the en- 
tropy of a system - composed of sub-systems A and B 
- equals the entropy of A plus the expectation value of 
the entropy of £?, conditional on A. If SK1-SK4 hold, 
the only possible entropy is BG 0, If only SK1-SK3 
hold (additivity axiom violated) Eq. (|2]) is the general- 
ized entropy with the constants (c, d) characterizing the 
universality class of entropy, (c, d) = (1,1) is the class 
of BG entropy, (c, d) = (q, 0) is the class of Tsallis en- 
tropies. A universality class (c, d) not only characterizes 
the entropy of the system completely in the thermody- 
namic limit, it also specifies its distribution functions. 
Many recently introduced generalized entropic forms ap- 
pear to be special cases of Eq. © 0. The associated 
distribution functions are 



W k [B(l- 
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with B = ex P ( i-^i -c)r )> anc ^ as one P oss it>le 

choice, r = (1 — c+cd) _1 , [l|. The function Wk is the &'th 
branch of the Lambert-W function, which is a solution of 
the equation x = W{x) exp(W(x)). Only branch k = 
and branch k = — 1 have real solutions Wk . Branch k = 
is necessary for all classes with d > 0, branch k = — 1 for 
d < 0. The generalized logarithm for the entropy Eq. (j2j) 
is the inverse of £ = A -1 . Further properties of systems 
where SK1-SK3 hold are reported in [4]. 

It has often been argued that for statistical systems 
with strong and long-range correlations, Boltzmann- 
Gibbs statistical mechanics loses its applicability, and 
that under these circumstances generalized entropies be- 
come necessary. This is certainly not true in general. 
While correlations can be the reason for non-Boltzmann 
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distribution functions, BG entropy often remains the cor- 
rect extensive entropy of the system [5[- 

In this paper we clarify the conditions under which BG 
entropy breaks down as the extensive entropy of a system. 
For ergodic systems, covering phase space, BG is always 
valid, regardless of what the correlations in the system 
might be. This was explicitly shown for binary systems 
in [5j. It is obvious that the structure of phase space, i.e. 
Gibbs T-space, is responsible for the Boltzmann-Gibbs 
framework to collapse and for generalized entropies to be- 
come necessary. Here we show that mere non-ergodicity 
is not enough: for generalized entropies to become nec- 
essary, T-space has to collapse in a specific way. 

In the following we derive all results in terms of growth 
of phase space volume as a function of system size. We 
illustrate our results for binary systems where a graph- 
ical representation is possible in terms of decision trees. 
Binary systems with correlations [6|, [7[ have been studied 
in the light of generalized entropies in Isl-Hl) . On the 
basis of growth of T-space as a function of the number of 
states we present a set of concise criteria when general- 
ized entropies are unavoidable and specify them by their 
universality classes. 

What does extensivity mean? Consider a system with 
N elements, each of which can be in one of m states. 
The number of system configurations (microstates) are 
denoted by Q(N), which depends on N in a system- 
specific way. Starting with Eq. (pQ) for equi-distribution, 

Pi = l/Vt (for all z), we have S g = J2?=i d(Pi) = ^(l/fi). 
Extensivity for two subsystems A and B means that 

n A+B g (1/SIa+b) = V A g (l/Sl A ) + Q B g (l/Sl B ) • (4) 

Using the primary scaling property of generalized en- 
tropies lim a ^ + 9 g(x) = ( see Q)> we & e t asymptoti- 
cally g'(x) = cg(x)/x, and Eq. (j4]) becomes 

g'(l/n A+B )=g'(l/n A )+g'(l/n B ) . (5) 



The derivative of g is the generalized logarithm, g'(x) 
— A(x), and 



1 



£ 



A 



1 



A 



1 



i 



i 



A generalized product <S) g can now be defined as x <S> g 
y = £ [A(x) + A(y)]. If each 'particle' can be in one of 
m states, we finally get for the number of states in the 
system 
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or if we use the distribution functions and generalized 
logarithms of generalized entropies, Eq. (j3j), the number 
of microstates grows asymptotically as 



n(N) 



i 



£cMc-l)N) 



exp 



d 



Wi 



k (M 



1 



c)N* 
(8) 



y 1 V 

r 3/ 1 \ r 3/ 2 \ r 3 



r N/ \ f N 



FIG. 1: Decision tree (triangle) for binary processes repre- 
senting the probabilities r^ of n heads and N — n tails af- 
ter N throws, r^ is the probability of a specific sequence, 
{(pi, (f2, . . . , <£at}, which does not depend on the order of 
events, but only on the number of n events in state ipi = 1 and 
N — n events in state <pj = 0. Leibnitz rule (scale invariance) 



8] holds if < + C+i 



_1 at all levels N and N - 1. 



where \i is some positive constant. At this stage note that 
for all non-BG systems, i.e. (c, d) =^ (1,1), the number 
of states Q(N) grows sub-exponentially with N. 

Inversely, given the phase space volume as a function 
of system size, we can now compute the generalized en- 
tropy, i.e. its universality class characterized by (c, d). 
Again using the primary scaling property for generalized 
entropies and de L 'Hospital rule A c = liuL c ^ + 9 g(x) ~ 
= ^Sf ' to S ether with E( l- © we get for large 

Sl(N) 



A C_1 A (l/fi(JV)) = A (X/Sl(N)) 



(9) 



or A c -! = limAr^oo •> wn i cn can be simplified to 



1 1 . „W(N) 
- = lim N—j—^- 

l-C at^oo f](7V) 



(10) 



For d we use the secondary scaling relation for general- 
. (6) ized entropies Q, (1 + a) d = lim^o 9 }a Cg ( x ] • Taking the 



derivative with respect to a on both sides we get 



(X 1+a ) 



d(l + a) d_1 = (l + a) d lim log x . , , , 

v v x^o \ g(x 1+a ) 

(ii) 

Set a —> 0, and use de L 'Hospital rule to get d = 
lim £C ^o(l — c + xg rr {x) / g r (x)) so that with Eq. dJJ we 
have 



/id 

d = lim log Q — — + c - 1 



(12) 



To see how phase space collapses for generalized en- 
tropy systems we illustrate the above result in the con- 
text of binary sequences, e.g. ip = {0, 1, 1, 0, 0, 1, • • • }. 
Let (p\ N = {tpi}f =1 denote a sequence of length TV. 
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Any correlations in sequences (on all levels of N) @ 
are completely determined by the set of joint probabil- 
ity functions {pn}n=h wnere Pn( ( Pi, ( P2, • • • ,^iv) is the 
joint probability of a sequence of length N. A sequence 
(p\ N contains k(N) = 'ones' and N — k(N) 'ze- 

ros'. If pn(<Pi, <P2, • • • , <Pn) is totally symmetric in its 
arguments the probability of sequences of length N de- 
pends on k only, = pjv(<£i> • • • ? <Pn)- There exist 
(^) sequences <p\ N with k 'ones' and N — k 'zeros' and 

^2k=o { I k) r k ~ 1- The can ^ e arranged into a trian- 
gle, Fig. [TJ representing the equi-probable sequences of 
binary events in a 'decision tree'. The full phase space 
volume (on level N) is 



- N 



{0,1} 



N 



n(N) 



N\ 



In other words, the number of microstates (number of 
sequences up to level N) is Q(N) = 2 N . Obviously the 
number of states increases exponentially and when sub- 
stituted in Eqs. ([TO]) and ([12]) we recover (c, d) = (1, 1), 
i.e. BG entropy. 

We now introduce restrictions to phase space such 
that not all sequences are allowed on all levels TV any- 
more. We denote the number of states in the restricted 
phase space by ft( fi )(iV) = |rj^|, and can immediately 
discuss an interesting fact. Imagine that phase space of 
sequences is extremely confined, say to a situation where 
in the thermodynamic limit all sequences approach a 
common point (same number of 'ones' in the sequence) 
£ = lmiAr^oo k(N)/N. In all cases where this point £ is 
neither or 1, BG is the only possible extensive entropy. 
This can be formulated in a 



(R) 



Theorem: Define a restricted phase space T N 
level N by 



on 



■ N 



N 

<p\ N eT N : K(N)<J2^<K(N) 



i.e., at level N the system only allows sequences 
cp\ N with more than K(N) and less than K(N) 
'ones'. If the restriction of phase space is such that 

liniAr^oo lim^oo = f, where £ G (0,1), 

then asymptotically the number of states grows expo- 
nentially (Q(N) = b N for some b > 0), and the extensive 
entropy is BG. 

The proof is to show that both, lower and upper 
bounds for fl(N), yield BG. The theorem states that 
for generalized entropies to exist it is necessary that the 
sequences are constrained to the situation where either 

limyv^oo K ^ = 0, or liniTv^oo ~jv^ = 1. In other words 
the sequences are asymptotically confined to a region of 
measure zero around the flanks of the decision triangle, 
i.e. the boundary of phase space. The theorem has two 
further implications: 

1. In case of probability distributions pn which are 
not totally symmetric in their arguments, generalized en- 
tropies can exist even though phase space need not be 





FIG. 2: Binary decision trees: (a) Schematic view of allowed 
sequence regions. If sequences are confined to the shaded 
regions (left of critical sequence line k crit ) the extensive en- 
tropy of the system is not BG but a generalized entropy, (b) 
Maximum line k crit at which BG entropy starts. Any system 
containing this line or sequences to the right of it, will be BG. 
(c) If the region is confined to a strip of size b, the extensive 



(13) entropy is Tsallis entropy, S q ,o, with q = 1 



l 

b • 



limited to the boundary of the decision triangle (as in the 
theorem). If the number of sequences ip G (i.e. the 
number of free decisions) up to level N grows sufficiently 
sub-linearly with AT, then the limit-points of sequences 
may be found along the entire base of the decision tri- 
angle (compare remark on super-diffusive random walks 
below). This means that the multiplicity of sequences 
with k out of N 'ones' in the large N limit grows suffi- 
ciently slower than the binomial multiplicity for totally 
symmetric pjy. 

2. The theorem can trivially be generalized from bi- 
nary processes to ra-state systems. This is done by pass- 
ing from the binomial to a multinomial description. 

We now show how different restrictions on phase space 
lead to various specific generalized entropies. We assume 
the existence of a critical sequence (p crit which follows the 
path k crit (N), see Fig. [2]a. This means that after N steps 
the sequence has produced a maximum of k crit (N) 'ones'. 
To the right of this sequence all sequences are forbidden. 
The phase space volume of such systems grows like [HI 



k crit (N) , 

Q(*)(iV)= £ ( 

i=l ^ 



(14) 



For any k, lim^oo J2i=i Ci) / Ck) = ^ which allows to 
asymptotically approximate Eq. (fT4|) 



(15) 



Using Stirling's formula, taking logs on both sides and 
keeping terms to leading order we arrive at 



k crit (N) w ATexp 



1 

"TV 



\ogQ(N) 



(16) 



This means that for any system whose sequences are 
confined to regions left to the critical sequence k crit (N), 
generalized entropies as specified in Eq. (j8j) are neces- 
sary. We now discuss some examples. 
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Maximum restricted phase space. Consider 
(c,d) = (1,1), i.e. Q(N) = 2 N . From Eq. (USD we get 
k crit (N) « TV. This means that for systems with general- 
ized entropies /c crit (iV) grows in a sufficiently sub-linear 
way with iV, e.g. k crit (N) oc iV a with < a < 1. If 
a = 1 and k crit (N) — eN no matter how small £ > 0, 
the system belongs to BG. 

Power-law growth. For a power-like growth of 
phase space, Q(N) = TV 6 , we have /c crit (7V) « 
TV exp[W_i (-^ log — log 6)] . Expanding the Lambert- 
W function we get k crit (N) « frexp(- ) 6, in 

the large TV limit. The phase space collapse is seen in the 
decision triangle as a restriction to a strip of width 6, Fig. 
[2]c. In this case (c, d) = (1 — ^,0), i.e. Tsallis entropy 
applies exactly. This is a well known result [H, [l2| . 

Streched exponential growth. For stretched 
exponential growth Q(N) = exp(A7V 7 ), Eq. ([16]) 
can be rewritten to k crit (N) w logft/W-i[- logft/N] 
and the Lambert-W term is reasonably approxi- 
mated by log(iV/logfi) + log(log(iV/logfi)). With 
this k crit (N) « 1^ iog(iV) ' anc ^ en ^ ro Py i s 

(c,d) = (1,1/7). 

Note that systems with confined areas in their decision 
trees are examples for strong memory. The system has to 
remember how many 'ones' have occurred in its trajec- 
tories, see examples in [H, [To|- Inversely, given a critical 
sequence line k crit (N), the universality class of the cor- 
responding generalized entropy (c, d) can be computed. 

Given the dependence of phase space volume Vt on sys- 
tem size we showed how to determine the associated ex- 
tensive generalized entropy by computing the exponents 
(c, d). We demonstrated that different generalized (non- 
additive) entropies - i.e. (c, d) ^ (1,1) ~ correspond 
to different ways of sub-exponential growth of T-space. 
We related the growth of phase space volume to the in- 
crease of the number of statistically relevant (dynamical) 
micro-states in the system. We found that whenever the 
fraction of dynamical variables vanishes for large N 
limAr^oo 77 = 0, generalized entropies become unavoid- 
able. This extreme confinement of relevant variables to 
a set of measure zero means that almost all states in the 
system are the same, or equivalent ly, the bulk of the de- 



grees of freedom is frozen. In other words, statistically 
relevant activity happens within a tiny fraction, of 
the system which can be seen as a collapse of phase space 
volume to some low-dimensional 'surface'. 

In conclusion we hypothesize that generalized entropies 
are relevant for physical systems being dominated by 
'surface effects', including the following: 

• Self organized critical systems. In sandpiles consider 
discrete sites where sand grains can be. The (binary) 
state of a site is being occupied by a grain or not. In 
a sandpile the bulk of the system is occupied and just 
the surface of the pile contains its statistically relevant 
degrees of freedom. The trajectory of a sand grain in 
a classical sandpile model follows sequences much alike 
those in the decision tree, Fig. [2]c. 

• Spin systems with dense met a- structures, such as spin- 
domains, vortices, instantons, caging, etc. If these meta- 
structures bind a vast majority of spins into (met ast able) 
objects, the remaining spins - not belonging to these 
structures - can move freely only in surface-like regions 
between these objects. For instance spin systems on 
random networks growing with constant connectedness 
(number of links divided by number of nodes squared) 
can be shown to require Tsallis entropy. 

• Super-diffusion. Consider a one dimensional accel- 
erating random walk, where each left-right decision is 
followed by N@ (0 < f3 < 1) steps in that direction 
(TV being the total number of steps the walk has so 
far taken). This process is a super-diffusive process 
((x 2 (t)) ex t 2 ~P) which requires a generalized entropy of 
type (c,d) = (1,1/0). 

• Anomalous diffusion. The presented results could also 
apply whenever states of a statistical system are excluded 
by the presence of other materials restricting mobility 
in Euclidean space. Think e.g. of diffusion in porous 
media where statistically relevant action takes place on 
restricted surface-like areas, and not in full 3D. 

For non-commutative variables alternative routes to 
generalized entropies may exist [l3|, G3 • 
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[1] R. Hanel, S.Thurner, Europhys. Lett 93, 20006 (2011). 
[2] C.E. Shannon, The Bell System Technical Journal 27, 

379,623 (1948). 
[3] A.I. Khinchin, Mathematical foundations of information 

theory (Dover PubL, New York), (1957). 
[4] R. Hanel, S. Thurner, M. Gell-Mann, Proc. Nat Acad. 

Sci. doi/10.1073/pnas.H03539108 (2011). 
[5] R. Hanel, S.Thurner, C. Tsallis, Eur. Phys. J. B 72, 263- 

268 (2009). 

[6] B. de Finetti, Atti della R. Academia Nazionale dei Lincei 
6, Memorie, Classe di Scienze Fisiche, Mathematice e 



Naturale, 4 251-299 (1931). 

[7] Jaynes E.T., in Bayesian Inference and Decision Tech- 
niques with Applications, Goel P.K. and Zellner A., eds. 
(North-Holland, Amsterdam, 1986), p. 31. 

[8] C. Tsallis , M. Gell-Mann, Y. Sato, Proc. Nat. Acad. Sci. 
102, 15377 (2005). 

[9] J.A. Marsh, M.A. Fuentes, L.G. Moyano, C. Tsallis, 
Physica A 372, 183-202 (2006). 
[10] L.G. Moyano, C. Tsallis, M. Gell-Mann, Europhys. Lett 
73, 813 (2006). 

[11] A. Rodriguez, V. Schwammle, C. Tsallis, JSTAT, P09006 



5 



(2008). 

[12] Tsallis C, in Nonextensive Entropy M. Gell-Mann, C. 

Tsallis, eds., (Oxford University Press), (2004). 
[13] F. Caruso, C. Tsallis, Phys. Rev. E 78, 021102 (2008). 
[14] A. Saguia, M.S. Sarandy, Phys. Lett. A 374, 3384-3388 

(2010). 



[15] For m states the critical number is obtained similarly: 

n (H) (JV) = E £'w E|n|= . with the multi . indices 

n — (m , ri2 . . . , n m -i) and n* = (m , n2, . . . , n m _i , iV — 
|n|), with |n| = Y^=i^ n i- The multinomial factor is 



